The present invention relates generally to systems for digital color imaging, and, more particularly, relates to apparatus and methods for providing efficient color processing capability for digital video cameras based on focal plane arrays (FPA). The invention can be used in digital camera systems that are designed to replace conventional silver halide film cameras.
Many high speed motion events are filmed with high speed 16 mm film cameras for motion analysis and diagnostics. Typical frame rates for these cameras range from 500 to 1000 frames per second, and the shutter exposure times are usually less than 500 microseconds. Typical high speed events include airborne weapons separation testing, missile tracking, automotive and aerospace crash testing, manufacturing operations analysis, and others.
Silver halide film, however, requires time-consuming wet chemical processing and handling. Consequently, considerable time elapses before the image results are available for analysis. In addition, film processing requires the use of chemicals requiring storage and disposal. This chemically-intensive process adds to the overall cost of film-based systems.
Further, the film must be optically scanned with a film scanner to convert the image sequence to digital form for computer-aided motion analysis. This extra step can be unacceptably time-consuming.
High speed digital video camera systems have been developed to replace conventional silver halide film cameras. These cameras produce images in digital form that can be stored on digital media (e.g. hard disk, magnetic tape, compact disk, etc.) and analyzed on a computer. These systems provide immediate access to image data, thereby circumventing time-consuming film processing steps.
Conventional high speed digital camera systems are based on FPAs, such as charge-coupled devices (CCDs), which sense broadband visible radiation in the 400-730 nanometer wavelength range. Although monochrome performance can be satisfactory for certain applications, color capability is required for several reasons.
For example, although monochrome digital cameras are suitable for high speed motion analysis of targets that exhibit significant luminance contrast, they cannot distinguish different color objects having similar luminance characteristics.
Color sensing capability can also provide the contrast necessary to distinguish key features in an image, even under low illumination conditions. A color camera thus provides important target tracking and identification capabilities.
For example, identification of different components or fragments can be difficult when using monochrome cameras for impact testing or weapon release testing. By color coding the object(s) under test or using existing color differences, color cameras can be used to readily identify the source of fragments or components during the course of the test.
Further, qualitative evaluation of monochrome video sequences by human observers can be difficult because natural color cues are absent. Routine subjective evaluation of images from color cameras can be performed more efficiently because color images provide human observers with a more realistic reproduction of the test scene. Color cameras also enable interpretation of physical phenomena, such as plasma plume temperatures, during weapons deployment.
Accordingly, many tests and applications which can be undertaken with the use of full color images may be difficult or impossible with black and white images.
Nearly all digital color FPA cameras are based on the principle of color pre-filtering. Since FPAs sense radiation throughout the visible spectrum (400-700 nm), optical color filters must be placed in front of the FPA to sense color. There are several methods for accomplishing this: multiple FPA systems, color filter wheels, and color filter array (CFA) single-chip FPAs.
Multiple FPA configurations have been developed for high-end color FPA cameras. Although these cameras produce images with minimal artifacts and low noise, they are costly and bulky compared to cameras based on a single FPA. In addition, multiple FPA cameras generate as much as three times the amount of data produced by a monochrome camera, thus placing excessive data handling demands on the image storage hardware.
Certain single-chip FPA color cameras utilize a rotating color filter wheel. These cameras cannot be shuttered at high frame rates, due to FPA readout and filter wheel rotation rate limitations. Consequently, this method is substantially limited to still frame applications, and is not suitable for high speed camera applications.
A number of conventional FPA video camera systems provide color sensitivity by means of color filter arrays (CFAS) placed in front of, or deposited directly on, a single-chip FPA. xe2x80x9cAdditivexe2x80x9d CFA/FPA combinations, utilizing red, green, and blue (RGB) filter xe2x80x9cmosaics,xe2x80x9d have been in common use since the mid-1970""s. These mosaics attempt to match the wavelength-dependent sensitivity of the human eye by including a larger percentage of green pixels than red and blue pixels.
By way of example, reference is made to the following U.S. and European Patent Office patents and other publications, the teachings of which are incorporated herein by reference:
xe2x80x9cEnabling Technologies for a Family of Digital Camerasxe2x80x9d, Parulski and Jameson, SPIE Vol. 2654, pp. 156-163, 1996;
xe2x80x9cSingle-Chip Color Camera Using a Frame-Transfer CCDxe2x80x9d, Aschwanden, Gale, Kieffer, Knop; IEEE Transactions on Electron Devices, Vol. ED-32, No. 8, August 1985, pp. 1396-1491;
xe2x80x9cA New Class of Mosaic Color Encoding Patterns for Single-Chip Camerasxe2x80x9d; Knop and Morf; IEEE Transactions on Electron Devices, Vol. ED-32, No. 8, August 1985, p. 1390-1395;
xe2x80x9cColor Filters and Processing Alternatives for One-Chip Camerasxe2x80x9d; Parulski; IEEE Transactions on Electron Devices, Vol. ED-32, No. 8, August 1985, pp. 1381-1389;
xe2x80x9cColor Image Compression for Single-Chip Camerasxe2x80x9d; Tsai; IEEE Transactions on Electron Devices, Vol. 38, No. 5, May 1991, pp. 1226-1232;
xe2x80x9cColor Imaging System Using a Single CCD Arrayxe2x80x9d; Dillon, Lewis, Kaspar; IEEE Transactions on Electron Devices, Vol. ED-25, No. 2, February 1978, pp. 102-107;
Hunt, R. W. G., Measuring Colour, Ellis Horwood Limited, 1995.
In systems using conventional, additive CFA/FPA devices, before the user can view the captured images in color, the image data must be xe2x80x9cdecoded.xe2x80x9d In the decoding process, mathematical interpolations are executed to recreate three full-resolution color xe2x80x9cplanesxe2x80x9d (the red, green and blue (RGB) planes common to conventional, additive CFA systems) from the original image. A number of color decoding schemes have been developed to recreate the full RGB color planes from an RGB CFA plane. These processing methods are employed in some still-frame point-and-shoot color CCD cameras and 35 millimeter digital camera back products.
However, conventional xe2x80x9cadditivexe2x80x9d CFA imaging systems using RGB sampling and processing suffer from significantly reduced light sensitivity, compared to monochrome FPA imaging systems. The sensitivity of a conventional RGB single-chip color FPA camera is significantly less than its monochrome counterpart due to absorption of light in the red, green and blue elements of the CFA. By way of example, the transmission efficiencies of conventional red, green, and blue filters (assuming tungsten-halogen lamp incident illumination) are 50%, 15% and 15%, respectively. Sensitivity is typically not a critical issue for still-frame cameras, because the corresponding exposure times are relatively long (on the order of 5 milliseconds). However, it is an important design issue for high speed cameras, because the exposure times are relatively short (less than 1 millisecond). Further, as the illumination on the pixels is reduced, the effect of noise becomes more significant. For these reasons, RGB xe2x80x9cadditivexe2x80x9d CFAs are not suitable for operation under low-luminance conditions, particularly for high-speed cameras.
Accordingly, there exists a need for digital color imaging systems with improved sensitivity, capable of capturing high-speed motion events under less-than-optimal illumination conditions for subsequent review and/or analysis.
It is accordingly an object of the invention to provide improved digital color imaging methods and apparatus affording efficient color processing capability for digital FPA cameras.
Another object of the invention is to provide the capability of adding efficient color imaging and signal processing to existing FPA camera devices with only minor modifications to the camera system.
A further object of the invention is to provide such methods and apparatus having increased computational efficiency and speed, while maintaining high sensitivity to light, limited color aliasing, and enhanced chromatic and spatial reproduction quality.
Still another object is to provide color imaging and processing without significant increases in data throughput rates and interim storage requirements as compared with current monochrome CCD-based imaging systems.
Other objectives include the provision of high overall performance in a single-chip FPA/CFA system, and the maintenance of high spatial resolution and signal/noise ratios (SNR) when compared with monochrome imaging systems.
The invention is also intended to enable simplified fabrication of FPA/CFA combinations; simplified hardware requirements for full color reconstruction; and rapid, simplified reconstruction requirements for black and white imaging when desired.
Other general and specific objects of the invention will appear hereinafter.
The foregoing objects are attained by the invention, which provides improved FPA color imaging systems and processing methods.
One practice of the invention involves the step of sampling an image, using an FPA camera equipped with a cyan, yellow and white (CYW) color filter array (CFA) to obtain a value for a given color at each pixel position, thereby creating a set of values sampled on a per-pixel basisxe2x80x94i.e., a compressed dataset representing the color image. The set of pixel values thus sampled has, for each pixel position in the CFA/FPA, a single sampled value, corresponding to the color sampled by the CFA/FPA at that pixel position. The data compression inherent in this method results from the fact that the set of sampled pixel values contains, for each pixel position, only one xe2x80x9ccolor value,xe2x80x9d rather than three.
In accordance with this practice of the invention, single-color cyan, yellow and white (CYW) color planes are generated from the set of sampled pixel values. A interpolating digital image signal processor then interpolates among and between the color planes to xe2x80x9cfill inxe2x80x9d the xe2x80x9cgapsxe2x80x9d in the individual color planes. The signal processor then transforms the interpolated CYW values from the single-color planes into RGB values which can be stored, transmitted, or utilized to drive a conventional video monitor or color printing device.
In accordance with a further practice of the invention, the interpolating processor executes a series of weighted summation steps that incorporate the sampled value at each pixel, as well as the values of the pixels surrounding the pixel to be interpolated.
In a further practice of the invention, the signal processor executes filtering and other post-interpolation steps to reduce color aliasing. The RGB color values at each pixel are then calculated from these filtered color planes. In particular, the processor can utilize a square xe2x80x9cwindowxe2x80x9d with weighting coefficients to filter the interpolated C,Y,W color planes before color is estimated, thereby reducing the spatial bandwidth of these planes. Then, after the C, Y, and W planes are filtered, a color look-up table (LUT) is used to determine the corresponding red, green, and blue values at each pixel location.
In a further preferred practice of the invention, since the white plane is an accurate representation of a monochrome luminance image, the white plane is used to introduce the high frequency content back into the filtered color image. This is accomplished by multiplying each pixel in the fully-generated red, green and blue planes by the white plane value, normalized by a linear combination of the red, green and blue values at that pixel position. The luminance at each pixel is thereby adjusted so that it is equal to that of the white plane value at that location.
In a further preferred practice of the invention, the signal processor utilizes a color LUT to transform the CYW values to RGB values. The RGB planes are subsequently spatially filtered before white plane multiplication.
In yet a further preferred practice of the invention, the signal processor transforms the CYW values to CIELAB color space values. The interpolated W plane is used to calculate L* values, and C, Y, and W planes are used to calculate a* and b* values. The a* and b* planes are filtered using a window with weighting coefficients, and the CIELAB color space values are then transformed to red, green, and blue color planes using a color look-up table (LUT). In this case, the luminance of each pixel is directly related to the W plane value at that pixel location.
The invention further encompasses software and hardware for executing these process steps, as disclosed hereinafter.
The invention will next be described in connection with certain illustrated embodiments; however, it should be clear to those skilled in the art that various modifications, additions and subtractions can be made without departing from the spirit or scope of the claims.